Use of the Probability Integral Transformation to Fit Nonlinear Mixed-Effects Models With Nonnormal Random Effects

This article describes a simple computational method for obtaining the maximum likelihood estimates (MLE) in nonlinear mixed-effects models when the random effects are assumed to have a nonnormal distribution. Many computer programs for fitting nonlinear mixed-effects models, such as PROC NLMIXED in SAS, require that the random effects have a normal distribution. However, there is often interest in either fitting models with nonnormal random effects or assessing the sensitivity of inferences to departures from the normality assumption for the random effects. When the random effects are assumed to have a nonnormal distribution, we show how the probability integral transform can be used, in conjunction with standard statistical software for fitting nonlinear mixed-effects models (e.g., PROC NLMIXED in SAS), to obtain the MLEs. Specifically, the probability integral transform is used to transform a normal random effect to a nonnormal random effect. The method is illustrated using a gamma frailty model for clustered survival data and a beta-binomial model for clustered binary data. Finally, the results of a simulation study, examining the impact of misspecification of the distribution of the random effects, are presented.